A deterministic gradient-based approach to avoid saddle points

Abstract Loss functions with a large number of saddle points are one the major obstacles for training modern machine learning (ML) models efficiently. First-order methods such as gradient descent (GD) usually choice ML models. However, these converge to certain choices initial guesses. In this paper, we propose modification recently proposed Laplacian smoothing (LSGD) [Osher et al., arXiv:1806.06317 ], called modified LSGD (mLSGD), and demonstrate its potential avoid without sacrificing convergence rate. Our analysis is based on attraction region, formed by all starting which considered numerical scheme converges point. We investigate region’s dimension both analytically numerically. For canonical class quadratic functions, show that region mLSGD $\lfloor (n-1)/2\rfloor$ , hence it significantly smaller than GD whose $n-1$ .